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BACKGROUND OF THE INVENTTON 



1. FIELD OF THE INVENTION 

The present invention relates generally to event processing, and in particular to associating 
the color of an object with an event. 

Portions of the disclosixre of this patent document contain material that is subject to 
copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone 
of the patent document or die patent disclosure as it appears in the Patent and Trademark Office 
file or records, but otherwise reserves all copyright rights whatsoever. 

2. BACKGROUND ART 

When using a computer or other interactive device a user can interact with the device in 
various ways. One way is to use a mouse. The user utilizes the mouse to position a pointer 
somewhere on a screen. The user then depresses a mouse button and depending on where the 
pointer is, something pre-programmed will happen if that area of the screen is "hot". 

In the context of the Intemet, the user can position the mouse pointer over a hot spot 
"hyperlinked" to a web page and depress the mouse button. This will cause the computer, set- top 
box, or other device to send a signal to a server to retrieve that web page and display it on the 
computer screen. 
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We foresee the desirability to mteract in this manner in many different contexts, not only 
linking with web pages. Current schemes are limited in the contexts that a user may interact. Before 
further describing the limitations of current schemes, a discussion of event processing is instructive. 

Event Processing 

"Event processing" refers to the association between some type of "action" and the 
occixrrence of an "event". Such a term is used herein in the context of a user's computing 
experience. The user performs an action followed by an event, followed by a series of other actions 
followed by other events. 

An action occurs when a user positions a cursor in a specified location on a computer 
screen, for instance using a mouse, and depresses one of the mouse buttons. This example shows a 
common type of action, but many other actions exist as well within the context of computer event 
processing, including the input of text, a voice command, or any other type of action. 

The example action may take place in any computing environment, but one common 
environment that the action occurs is in a web browser. A web browser is a graphical user interface 
that enables xisers to utilize the Internet, for instance by viewing web pages. Normally the user will 
position the movise pointer over a "hyperlink" and depress the mouse button there. This, in turn, 
triggers an event. The event in this example is to cause a server to retrieve another web page. Thus, 
in this example, the user interacts with the Intemet by a series of actions followed by events. 
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Current event processing is disadvantageous, in part, because the types of actions available 
and the events initiated by the actions are limited. Before further discussing the drawbacks of 
current event processing, an overview of the Internet is provided below. 



Internet 

The information network known as the "Internet" has become increasingly popular. The 
Internet provides a body of information which may be accessed by users via computers for such 
■C purposes as business, education, and entertainment. In particular, the Internet includes a lai^e 
^; collection of interconnected documents, images, sounds, videos, and other forms of media which 
are stored in computers in a system known as the World Wide Web or simply "the web". 

;:3 Intemet resources are typically accessed in a two-way enviroimient via a networis 

y connection. For example, connectivity to the network may be via a conventional twisted-pair 
G telephone line which has a relatively low data-carrying capacity (e.g., bandwidth), or via a higher 
bandwidth path, which may comprise optical fiber, coaxial cable, ISDN, xDSL, wireless 
connections, or other transmission mechanisms. To retrieve data from a network destination (e.g., 
an Intemet site), a user sends a message to the network destination to request the transfer of 
information to the user's computer, referred to as a client. The network destination typically includes 
a computer known as a server. The server then sends a request s^al to a source function, which 
maybe a memory which is coupled to the server. The source function includes the user-rcquested 
information which may comprise, for example, text, graphics, audio and/or video data. In response 
to the request signal from the server, the source function provides the requested information to the 
client. 



LA 35093v4 



4 



HTML 



Source information which is stored in the source function is often stored in a format known 
as "Hypertext Markup Language (HTML)". This file or script format allows the display of text, 
graphics and audio information, and provides links to other pages of information through 
"hyperlinks." Hyperlinks are strings of characters in a particular format that specify the address of 
the desired page of information. 

HTML is a system for marking documents to indicate how the document should be 
displayed, and how varioias documents should be linked together. HTML is a form of Standard 
Generalized Markup Language (SGML), defined by the International Standards Organization. 
HTML specifies the grammar and syntax of markup tags which are inserted into a data file to define 
how the data will be presented when read by a computer program known as a "web browser" . 
Conventional web browsers include Internet Explorer, Netscape Navigator, and others. 

The data file, which is typically stored on a server, includes one or more web pages which are 
visited by users who have computers which may run different browsers. When a page is visited, 
HTML data output from the server is downloaded to the client computer. The client computer's 
browser processes the data to format a layout for the page so the page may be viewed by the user on 
a computer screen. GeneraEy, HTML tags provide text formatting, hypertext links to other pages, 
and links to sound and image elements. HTML tags also define input fields for interactive web 
pages. 
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An HTML application is made available to users on the web by storing the HTML file in a 
directory that is accessible to a server. Such a server is typically a web server which conforms to a 
web browser- supported protocol known as Hypertext Transfer Protocol (HTTP). Servers that 
conform to other protocols such as the File Transfer Protocol (FTP) or GOPFIER may also be 
used, but do not support interactive HTML files. 

HTTP defines a set of rules that servers and browsers follow when communicatii^ with 
each other. Typically, the process begins when a user accesses an icon in an HTML page which is 
the anchor of a hyperlink, (for instance, by positioning a cursor on the icon and depressing a mouse 
button), or the user inputs a Uniform Resource Locator (URL) to his or her web browser, described 
below. A connection is then made to the server at the address and port number specified by the 
URL. Next, the browser sends a request to retrieve an object from the server, or to post data to an 
object on the server. The server sends a response to the browser including a status code and the 
response data. The connection between the browser and server is then closed. 

URL 



A URL is a unique address which identifies virtually all files and resources on the Internet. A 
URL has the form: 

method:/ / servenport/ path/f ile# anchor. 

The "method" of accessing the resource is the web browser-supported protocol, and may include, 
for example, HTTP, FTP, GOPHER, TELNET, NEWS, orMAILTO. The "servenport" indicates 

6 
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the name of the server \diich is providing the resoiirce, and is alternatively known as the Internet 
domain name. For example, many businesses will use their business name as part of the server field. 
The port designation is the port number on the server, but is usually not used since a default port is 
assumed. The "path" indicates the directory path to the resource. The file indicates the file name of 
the resource. The "anchor" indicates the named element in the HTML document. Not all fields are 
required. 

Consider the following URL, for example: 

http:// www.business.com/information/ profits.html# section_L 

The access method is HTTP, the server is www.business.com, there is no port specified, the path is 
news, the file is profits.html, and the anchor is section_l. 

Thxis, it maybe seen that web browsers operate in a two-way communication environment 
to access information by sending a request signal defined by a URL command to a server, and 
receiving information in return. 

Current Event Processing Schemes 

In the context of the Internet, event processing has been limited by the nature of the web. 
Originally, the web was largely a textual body of intercoimected information. Actions were limited 
to depressing a mouse button on a text link which initiated the event of retrieving another web page 
from a specified URL. These methods are disadvantageous because they are static. Such events 
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have no application in an active environment, for instance where streaming video or television 
signals are involved, and such events have no application outside of a web browser (i.e., where a 
GUI is tianslating HTML commands into actions). 

As the web evolved, other types of event processing have emerged. In particular, the web 
evolved to have a more active nature. First, the text-based web became merged with images. The 
images resided in various formats and could also be used to associate an action with an event. The 
images, however, suffered the same drawback as the text and essentially performed the same 
function. For instance, the images were used in the same event context (e.g., depressing a mouse 
button on the image) and the same actions resulted (e.g., the image caused a server to send another 
web page to the user). 

After images came more active web envirormients. One example came in the form of 
animation where a sequential series of images are displayed in the web browser which cause the 
image to appear to move on the screen. To allow the user to act upon an animated image, one 
scheme takes each frame of the animated image and pre- determines where on that individual frame a 
potential action should be associated with an event. 

For instance, imagine an animation of a runner holding a baton where the runner is moving 
horizontally across the screen and the desired action is to depress a mouse button when the pointer 
is over the baton. In this example, assume the animation consists of 100 images. Using one 
scheme, the 100 images are each individually stored in a computer and each image is analyzed to find 
where the baton was on the screen. 
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AloE^ with each of the 100 im^es is an image map specifyii^ that in frame 1 the boundaiy 
of the baton is in a first position and in frame 2, the boundary of the baton is in a second position. 
To map to these positions, the prior art essentially draws a wire frame around the baton in each 
frame. Therefore, 100 wite frames must be drawn in varying locations. The baton in a moving 
image varies in position and angle. Hiis in turn varies the shape of the wire frame in each frame of 
the image. 

The prior scheme is extremely disadvantageous because it requires a tremendous amount of 
manual pre-processing to draw 100 varied size wire frames. Such pre-processing contains a large lag 
on overhead. This scheme also must store each and every potential action in a computer and then 
wait for the action to occur. This technique uses a large amount of the computer's storage space 
and a large amount of time to create the wire frames. This scheme, in addition, is bound to a web 
browser implementation and may only trigger events relating to the accessing of a web page at a 
specified URL. 

As the web has developed more recently, such schemes have become even less 
advantageous. Specifically, the web has expanded away from a static model and now incorporates 
other technologies, such as television and other signals. The web through this expansion has also 
moved to other media forms, such as "set-top boxes" and interactive television sets where the user 
interacts with the web using a conventional television set as an output device. With these other 
media forms and other environments, other types of GUIs (not only web browsers) and other types 
of actions (not only the retrieval of a web page from a specific URL) are desired. However, there is 
currently no efficient -vray to process events in such environments. 
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SUMMARY OF THE INVENTION 



The present invention provides a method and apparatus for associating the color of an 
object with an event. In one or more embodiments, the invention takes a streaming or static digital 
image frames and associates an active, dynamic, or static object with an event by its color. 

In one embodiment, an event is the retrieval of a document at a specified URL and the 
display of the docimient in a web browser. In another embodiment, an event includes the retrieval 
of textual or audio information associated with the object. In another embodiment, the event 
comprises switching to a separate video stream in an environment using streaming media, such as a 
digital video disc (DVD) player or recorder, or other sviitable video environment. 

The invention defines the color in a location (termed a region of interest) by its digital color 
mathematical properties. For instance, one embodiment uses the RGB standard which separates the 
red, green, and blue properties of the color as nxmerical values. Another embodiment also xases the 
HSV characteristics of the color, which define hue, saturation, and value as components of color. 
Another embodiment uses both RGB and HSV characteristics. In one embodiment, the color 
characteristics are obtained from a digital RGB frame buffer 

As described above, the event links to another URL (i.e., to facilitate a transaction over the 
Internet) or in the case of a DVD or other video environment, the event causes the video stream to 
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transition to a different data track (i.e., to pull up different information) or to a different video 
stream. 

In one embodiment the invention uses an edge detection scheme to help define the 
boundaries for the colon in a digital RGB frame. The algorithm developed gives those pixels an 
"edge" which is not visible to the user, nor drawn on the screen, nor pre-processed prior to the user 
viewing the images formed by the pixels. The values of the pixels defines an edge rather than the 
actual drawing of a wire frame on the screen. 

Thus, one or more embodiments of the present invention create the associations between 
objects and events dynamically in that the data which creates the association is a property of die 
image itself (i.e., its color values). The present invention requires no alteration or pre-processing of 
the image to create the associations. 
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BRIEF DESCRIPTION OF THE DRAWINGS 



These and other feamres, aspects and advantages of the present invention will become better 
understood with regard to the following description, appended claims and accompanying drawings 
where: 

Figure 1 is a flowchart showing how to associate the color of an object with an event 
according to an embodiment of the present invention. 

Figure 2 is a flowchart showing how to defme the color characteristics of an object 
according to an embodiment of the present invention. 

Figiare 3 is a flowchart showing how to define the color characteristics of an object 
according to another embodiment of the present invention. 

Figure 4 is a flowchart showing how to define the color characteristics of an object 
according to another embodiment of the present invention. 

Figure 5 is a flowchart showing how to associate the color of an object with an event using a 
table according to an embodiment of the present invention. 

Figure 6 is a computer execution environment where one or more embodiments of the 
present invention maybe implemented. 

12 
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Figure 7 shows an embodiment of the present invention where the frame buffer is utilized. 



Figure 8 shows an embodiment of the present invention where color ranges are utilized. 
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DETAILED DESCRIPHQN OF TEiE INVENTION 



The invention is a method and apparatus for associating the color of an object with an event. 
In the following description, numerous specific details are set forth to provide a more thorough 
description of embodiments of the invention. It is apparent, however, to one skilled in the art, that 
the invention maybe practiced without these specific details. In other instances, well known 
features have not been described in detail so as not to obscure the invention. 

One or more embodiments of the present invention are directed to enhancing a user's ability^ 
to interact with a computer, set-top box, interactive television set, or other processing device. 
According to embodiments of the present invention, the user provides input on some area of a 
display device, for instance by depressing a mouse button at a specified location. The invention then 
determines where the viser has depressed the moiise button and what color value appears on the 
display device at the time the user has depressed the mouse button. Then, the invention initiates an 
event that corresponds to that color value. 

Event Processing 

In one or more embodiments, the invention takes a streaming or static media and associates 
an active, dynamic, or static object with an event by its given color. The event maybe, for instance, 
the retrieval of a document at a specified URL in a web browser or the retrieval of other information 
in non-web browser implementations. Other implementations include, for instance, the viewing of 
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mediaj via a television, DVD or other media player, set-top devices, web-TV, interactive TV, and 
video- games. 

Take, for example, a web page implementing streaming video. In the example, a game show 
host is wearing a green shirt. If a user provides input to the screen by depressing a mouse with the 
cxirsor in a location where the green shirt is located, the invention links to a URL associated with the 
green shirt. Thus, the moving color may substitute for a conventional, static, text link In another 
embodiment, the game show host with the green shirt may be in a television show. In this 
embodiment, providing input on the moving green shirt may initiate a different event, such as 
providing information about the shirt or the game show host. 

One embodiment of the present invention is shown in Figure 1. At step 100, an action 
occurs. At step 110, the location of the action (a region of interest) is determined. Locating a region 
of interest is weU known in the art. 

One method to locate a region of interest is to obtain an ordered pair corresponding to the 
location of the cursor at the moment of the action. For instance, a computer screen maybe 
represented as a two-dimensional graph wherein the axis of the graph resides in the lower left hand 
comer of the computer screen. Thus all ordered pairs will correspond to positive values of x,y pairs 
on the graph. 

After defining the region of interest, the value associated with a color in the region of 
interest is determined at step 120. Thereafter, at step 130, an event associated with that color is 
initiated. 
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Color Characteristics 



In one embodinaent, the invention defines the color by its mathematical properties. For 
instance, one embodiment uses the RGB standard which separates the red, green, and blue 
properties of the color as numerical values. Another embodiment also uses the HSV characteristics 
of the color, which define hue, saturation, and value as components of color. HSV characteristics 
further define the properties of a color, for instance V represents an analog of the intensity of a 
color or how bright the color appears. Another embodiment of the present invention uses both 
RGB and HSV characteristics. 

One embodiment defines color characteristics as shown in Figure 2. At step 200, it is 
determined whether an action has occurred. If no action occurred, step 200 repeats until an action 
occurs. If the action has occurred, it is determined where the action has occurred at step 210. 

The location is shown herein with respect to a single pixel, "pixel A". At step 220, a red 
component of a color for pixel A is determined. Then, at step 230, a green component of the color 
for pixel A is determined. Next, at step 240, a blue component of the color for pixel A is 
determined. Thereafter, at step 250, the color characteristics are combined to form a color value for 
pixel A. 

After step 250, it is determined whether any event is associated with the color value at step 
255. If no event is associated with that color value, no event is initiated and the process repeats at 
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step 200. If an event is associated with the color value, then at step 260 an event associated with the 
color value is initiated. 

One embodiment defines color characteristics as shown in Figure 3. At step 300, it is 
determined whether an action has occurred. If no action occurred, step 300 repeats until an action 
occurs. If the action has occurred, it is determined where the action has occurred at step 310. The 
location is shown herein with respect to a single pixel, "pixel B". At step 320, a hue component of a 
color for pixel B is determined. Then, at step 330, a saturation component of the color for pixel B is 
determined. Next, at step 340, a value component of the color for pixel B is determined. 

Thereafter, at step 350, the color characteristics are combined to form a color value for pixel 
B. After step 350, it is determined whether any event is associated with the color value at step 355. 
If no event is associated with that color value, no event is initiated and the process repeats at step 
300. If an event is associated with the color value, then at step 360 an event associated with the 
color value is initiated. 

One embodiment defines color characteristics as shown m Figure 4. At step 400, it is 
determined whether an action has occurred. If no action occurred, step 400 repeats until an action 
occurs. If the action has occurred, it is determined where the action has occurred at step 410. The 
location is shown herem with respect to a single pixel, "pixel C. At step 420, a hue component of a 
color for pixel C is determined. Then, at step 430, a saturation component of the color for pixel C is 
determined. Next, at step 440, a value component of the color for pixel C is determined. At step 
450, a red component of a color for pixel C is determined. Hien, at step 460, a green component of 
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the color for pixel C is determined. Next, at step 470, a blue component of the color for pixel C is 
determined. 

Thereafter, at step 480, the color characteristics are combined to form a color value for pixel 
B. After step 480, it is determined whether any event is associated with the color value at step 485, 
If no event is associated with that color value, no event is initiated and the process repeats at step 
400. If an event is associated with the color value, then at step 490 an event associated with the 
color value is initiated. 

Frame Buffer 

In a computer or other comparable device, images are often used as a form of output. A 
common way that such devices handle images or streams of images (e.g., video) is by using a "fmme 
buffer". For offline pre-processing or for real-time processing using high speed computers, the 
RGB frame buffers reside m memory. Alternatively, a special high speed video board can be used to 
process an RGB frame buffer for real-time processing. 

A video signal, for instance, comprises a series of still images shown sequentially where the 
series of images appears so quickly that a user perceives the series of pictures to be a continuous 
stream of motion. As the computer processes each itnage, data relating to the image is stored in the 
frame buffer. Such data includes, for instance, the color values associated with each pixel at each 
location on the screen. 
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As a new image is displayed, new data about the image is stored in the frame buffer. 
Depending on the size of the frame buffer, many sequential images maybe stored there before it 
fills up and old image data must be overwritten. 

One embodiment of the present invention takes advantage of the data akeady existing in the 
frame buffer, for instance to perfomi step 120 of F^e 1. Figure 7 shows an embodiment of the 
present invention where the frame buffer is utilized. 

At step 700, an action occurs. At step 710, the location of the action (a region of interest) is 
determined. Next, at step 720, the invention access the frame buffer to obtain color values at the 
location of the action when it occurred. Then, it is determined whether any event is associated with 
the color value at step 730. If no event is associated with that color value, no event is initiated and 
the process repeats at step 700. If an event is associated widi the color value, then at step 740 an 
event associated with the color value is initiated. 

Events 

As described above, one event is a link to another URL (i.e., to facilitate a transaction over 
the Intemet). Thus, one event may comprise die retrieval of a web page from a server computer 
connected to the Intemet. 

In the context of streamir^ media, an event maybe die retrieval of a text page. For instance, 
assume die user is watching a DVD program showing several of die great pyramids of Egypt. If the 
user causes an event to occur in a region of interest defined by a specific pyramid, then the event 
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may comprise the retrieval of a text page telling the user exactly which pyramid this is, when it was 
built, the history associated with this particular pyramid, and any other pertinent information. Thus, 
the event would cause the DVD to move to a different data track (i.e., to pull up different 
information). 



In the same great pyramid example, another event may comprise the retrieval of audio 
information, for instance a narrator providmg information similar to that given in the text track in 
the above example. Furthermore, the event may comprise the retrieval of a new media stream, such 
as the close up of the pyramid, alternate angles showing different views of die same pyramid, or 
views from inside the pyramid, for example. 

In one embodiment an event comprises the retrieval of a data track A data track comprises 
information associated with the color acted upon. Assume for instance, that this embodiment of the 
present invention takes place using an interactive television set. WhHe the interactive television set is 
operating, a game show host appears wearing a green shirt. Then assume, for instance, tiiat a user 
caused an action to occur at the location of the green shirt. In riiis instance, the data track may be 
information about the shirt, the game show host, or any otiier relevant information. 

Whether an event is associated with a color value maybe determined in a plurality of ways. 
One such way is to use a table. An example of one embodiment of a table is shown in table 1. 



Table 1 



Color Value 


Event 


100 


retrieve document at URL www.mfo.coni/test 
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300 


obtain data track at a specified location 


200 


initiate media stream at specified location 


288 


initiate audio track at specified location 



The table maybe arranged in a plurality of ways and maybe accessed in methods well 
known to those skilled in the art, includii^ hash tables, hash functions, pointers, indices, or any 
other suitable form of table lookup. In addition, different color values may initiate the same event. 
Alternatively, every color value on an output device maybe associated with the same event. 



F^e 5 shows an embodiment of the present invention where a table is implemented. At 
step 500, an action occurs. At step 510, the location of the action (a region of interest) is 
determined. Then at step 520, the value associated with a color in the region of interest is 
determined. Next, a table is searched to locate the color value to obtain an event that is paired with 
the color value at step 530. Then, at step 535, it is determined whether the color value is in the 
table. If it is not, then that color value is not linked to an event and no event is initiated. In that 
case the process repeats at step 500. Otherwise, at step 540, tlie event is initiated. 

Color Ranges 



In another embodiment of the present invention regions of interest (i.e., those regions where 
a user has performed an action) are associated with events based on color ranges. A color range, for 
instance, includes those shades of blue that correspond to the body of a bluebird in a video stream 
Realistic image and video streams will not have a uniform shade of blue at all areas of the bluebuxi's 
body since light and other factor will cause natural variations in the perceived color reflected from 
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an object. In this embodiment, this reality is accounted for, which gives this embodiment specific 
application in dynamic and high-quality video streams. 

Color ranges are defined by color characteristics. For instance, a color range may translate 
into values on an RGB or HS V scale, or could be combinations of both. Typically, m a computer 
system such values range from 0-255. Thus, the bluebird's body may be defined as a region of 
interest where the color range includes a B value on the RGB scale of 50-75, for instance. 

Thus, whether the action occurred at an upper left or lower right portion of the bluebird's 
body, the same event is initiated, despite the potential that at these corner of the regions of interest, 
the color values differed. All that is required in this embodiment is that both differing color values 
fall within the defined color range. 

In many images, the colors maybe very similar, yet they may be aspects of different 
components of the image and associated with different events. In one embodiment, the langes are 
defined with respect to a plurality of color characteristics including RGB and HS V. 

For instance, a range maybe defined where the R is selected to have a range of 50-80, the G 
is selected to have a range of 150- 160, and the B is selected to have a mnge of 200-220, An example 
color is selected by the user where the example color has RGB characteristics of 60,155, and 205 
respectively. Another color on the screen that is associated with a different event may appear to be 
substantially identical to the example color but has RGB characteristics of 190, 100, 33. Since the 
ranges have been defined by a plurality of characteristics, however, such minute differences in color 
shading are accounted for by this embodiment of the present invention. 
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In one embodiment, the characteristics of the color are defined by R, G, B, H, S, V, and R- 
G, and G-B, where R-G and G-B have red and green and green and blue components respectively. 
In this embodiment, any three of these variables is sufficient to uniquely define all colors displayed 
on an output device or stored in a frame buffer. To determine whether the variables are within the 
color range for the purpose of determining whether an event will be associated with the area of 
interest, the following algorithm is xised (shown in F^re 8). 

At step 800, it is determined whether an action has occurred. If no action occurred, step 800 
repeats until an action occurs. If the action has occurred, it is determined where the action has 
occurred at step 810. Then at step 820, three variables are chosen from a list of possible color 
components, for instance, R, G, B, H, S, V, R-G, or G-B. For the chosen variables, histograms are 
produced at step 830 

A histogram is a graphical method for describing data. It is a presentation of quantitative 
data (i,e., color components) that have been summarized in a frequency distribution. The values of 
the quantitative variable are shown on the horizontal axis, A rectangle is drawn above each class 
such that the base of the rectangle is equal to the width of the class interval and its height is 
proportional to the number of data values in the class. 

Using the histogram, the color ranges are determined at step 840, for instance by 
determining the variable peak on the histogram + or - three times its variance. After step 840, it is 
determined whether any event is associated with the color range at step 850. If no event is 
associated with that color range, no event is initiated and the process repeats at step 800. If an event 
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is associated with the color range, then at step 860 an event associated with the color range is 
initiated. 

Color Patterns 

It is often beneficial to define a regions of interest that is not uniform in color. A region of 
interest, for instance, maybe a plaid tablecloth. In this case, it is not adeqmte to use a single color 
value, nor a color range to define the region of interest. To account for this, one embodiment of the 
present invention uses color patterns to define the region of interest. 

A color pattern comprises a representation of the colors in the region of interest, for 
instance a graphical representation, defined by a plot on an x,y axis. The varying colors in the region 
of interest create peaks and valleys in the color pattern, graph representation. By locating patterns 
of peaks and valleys in such a color pattern, multi-color patterns, such as the plaid tablecloth, are 
used to associate actions with events. 

Edge Detection 

One embodiment of the present invention uses an edge detection scheme to define the 
boundaries for the colors. The color component of an object is described by its pixels. An 
algorithm gives those pixels an "edge" which is not visible to the user, nor drawn on the screen, nor 
pre-processed prior to the user viewing the images formed by the pixels. For instance, an invisible 
wire frame is created to describe the shape of the given colored object to associate with the event. 
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But the value of the pixels defines the wire frame rather than actually drawing a wire frame on the 
screen. 

One embodiment of the present invention uses Quicktime, a software package available 
from Apple Computer™ . In this embodiment, Quicktime is used to support the use of "sprites". 
A sprite is a clickable element withm a video stream Using Quicktime, a sprite is modified to be 
defined by its color component. Thus, one or more embodiments of the present invention create 
the associations between objects and events dynamically, in that the data which creates the 
association is a property of the image itself (i.e., its color values). The present invention requires no 
alteration or pre-processmg of the image to create the associations. 

Embodiment of Computer Execution Environment (Hardware^ 

An embodiment of the invention maybe in^lemented as computer software in the form of 
computer readable program code executed in a general purpose computing environment such as 
environment 600 illustrated in F^;ure 6, or in the form of bytecode class files executable within a 
Java™ run time environment running in such an environment, or in the form of bytecodes running 
on a processor (or devices enabled to process bytecodes) existing in a distributed environment (e.g., 
one or more processors on a network). Akeyboard 610 and mouse 611 are coupled to a system bus 
618. The keyboard and mouse are for introducing user input to the computer system and 
communicating that user input to central processing unit (CPU) 613. Other suitable input devices 
maybe used in addition to, or in place of, die mouse 611 and keyboard 610. 1/ O (input/output) 
unit 619 coupled to bi-directional system bus 618 represents such 1/ O elements as a printer, A/V 
(audio/ video) 1/ O, etc. 
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Computer 601 may include a comaminication interface 620 coupled to bus 618. 
Communication interface 620 provides a two-way data communication coupling via a network link 
621 to a local network 622. For example, if communication interface 620 is an integrated services 
digital network (ISDT^ card or a modem, communication interface 620 provides a data 
communication connection to the corresponding type of telephone line, which comprises part of 
network link 621. If communication interface 620 is a local area network (LAN) card, 
communication interface 620 provides a data communication connection via network link 621 to a 
==C compatible LAN. Wireless links are also possible. In any such implementation, communication 
:f: interface 620 sends and receives electrical, electromagnetic or optical signals which cany digital data 
streams representing various types of information. 

h Network link 621 typically provides data communication through one or more networks to 

Q other data devices. For example, network link 621 may provide a connection throiigh local network 
C' 622 to local server computer 623 or to data equipment operated by ISP 624. ISP 624 in turn 

provides data communication services through the world wide packet data communication network 
now commonly referred to as the "Intemet" 625. Local network 622 and Intemet 625 both use 
electrical, electromagnetic or optical signals which carry digital data streams. The signals through die 
various networks and the signals on network link 621 and through commimication interface 620, 
which carry the digital data to and from computer 600, are exemplary forms of carrier waves 
transporting the information. 

Processor 613 may reside wholly on client computer 601 or wholly on server 626 or 
processor 613 may have its computational power distributed between computer 601 and server 626. 
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Server 626 symbolically is represented in Figure 6 as one unit, but server 626 may also be distributed 
between multiple "tiers". In one embodiment, server 626 comprises a middle and back tier where 
application logic executes in the middle tier and persistent data is obtained in the back tier. In the 
case where processor 613 resides wholly on server 626, the results of the computations performed 
by processor 613 are transmitted to computer 601 via Internet 625, Internet Service Provider (ISP) 
624, local network 622 and communication interface 620. In this way, computer 601 is able to 
display the results of the computation to a user in the form of output. 

Computer 601 includes a video memory 614, main memory 615 and mass storage 612^ all 
coupled to bi-directional system bus 618 along with keyboard 610, mouse 611 and processor 613. 
As with processor 613, in various computing environments, main memory 615 and mass storage 
612, may reside wholly on server 626 or computer 601, or they may be distributed between the two. 
Examples of systems where processor 613, main memojty 615, and mass storage 612 are distributed 
between computer 601 and server 626 include the thin-client computing architecture developed by 
Sun Microsystems, Inc., the palm pilot computing device and other personal digital assistants, 
Internet ready cellular phones and other Internet computing devices, and in platform independent 
computing environments, such as those which utilize the Java technologies also developed by Sun 
Microsystems, Inc. 

The mass storage 612 may include both fixed and removable media, such as magnetic, 
optical or magnetic optical storage systems or any other available mass storage technology. Bus 618 
may contain, for example, thirty-two address lines for addressing video memory 614 or main 
memory 615. The system bus 618 also includes, for example, a 32-bit data bus for transferring data 
between and among the components, such as processor 613, main memory 615, video memory 614 
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and mass storage 612. Alternatively, multiplex data/ address lines maybe used instead of separate 
data and address lines. 

In one embodiment of the invention, the processor 613 is a microprocessor tnanufactured 
by Motorola, such as the 680X0 processor or a microprocessor manufactured by Intel, such as the 
80X86, or Pentium processor, or a SPARC microprocessor from Sun Microsystems, Inc. However, 
any other suitable n^iicroprocessor or microcomputer may be utilized. Main memory 615 is 
comprised of dynamic random access memory (DRAM). Video memory 614 is a dual-ported video 
random access memory. One port of the video memory 614 is coupled to video amplifier 616. The 
video amplifier 616 is used to drive the cathode ray tube (CRT) raster monitor 617. Video amplifier 
616 is well known in the art and maybe implemented by any suitable apparatus. This circuitry 
converts pixel data stored in video memory 614 to a raster signal suitable for me by monitor 617. 
Monitor 617 is a type of monitor suitable for displaying graphic images. 

Computer 601 may send messages and receive data, including program code, through the 
network(s), network link 621, and communication interface 620. In the Internet example, remote 
server computer 626 might transmit a requested code for an application program through Intemet 
625, ISP 624, local network 622 and communication interface 620. The received code may be 
executed by processor 613 as it is received, and/ or stored in mass storage 612, or other non- volatile 
storage for later execution. In this manner, computer 600 may obtain application code in the form 
of a carrier wave. Alternatively, remote server computer 626 may execute applications using 
processor 613, and utilize mass storage 612, and/ or video memory 615. The results of the execution 
at server 626 are then transmitted through Intemet 625, ISP 624, local network 622 and 
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communication interface 620. In this example, computer 601 performs only input and output 
functions. 

AppKcation code maybe embodied in any form of computer program product, A computer 
program product comprises a medium configured to store or transport computer readable code, or 
in which computer readable code may be embedded. Some examples of computer program 
products are CD-ROM disks, ROM cards, floppy disks, magnetic tapes, computer hard drives, 
servers on a network, and carrier waves. 

The computer systems described above are for purposes of example only. An embodiment 
of the invention maybe implemented in any type of computer system or programming or 
processing environment. 

Thus, a method and apparatus for associating the color of an object with an event is 
described in conjunction with one or more specific embodiments. The invention is defined by the 
claims and their full scope of equivalents. 
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CLAIMS 

1. A method for associating a color of an object with an event comprising: 
determining a location where an action has occulted; 

determining a color value for said location; 
initiating an event associated with said color value. 

2. The method of claim 1 wherein said color value comprises a color range. 

3. The method of claim 1 wherein said color value comprises a color pattern. 

4. The method of claim 1 wherein said color value comprises an RGB characteristic. 

5. The method of claim 1 wherein said color value comprises an FBV characteristic. 

6. The method of claim 1 wherein said location is a computer screen. 

7. The method of claim 1 wherein said event is the retrieval of a web page at a specified 

URL, 

8. The method of claim 1 wherein said event is the retrieval of a data track 

9. The method of claim 1 wherein said object is an image residing in a web browser. 
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10. The method of claim 1 wherein said object is an image residir^ in a streaming media. 



11. A computer program product comprising: 

a computer usable medium having computer readable program code embodied therein 
conf iguied to associate the color of an object with an event, said computer program product 
comprising: 

computer readable code configured to cause a computer to determine a location where an 
action has occurred; 

computer readable code configured to cause a computer to determine a color value for said 
location; and 

computer readable code configured to cause a computer to initiate an event associated with 
said color value. 

12. The computer program product of claim 1 1 wherein said color value comprises a 
color range. 

13. The computer program product of claim 1 1 wherein said color value comprises a 
color pattern. 

14. The computer program product of claim 1 1 wherein said color value comprises an 
RGB characteristic. 

15. The computer program product of claim 1 1 wherein said color value comprises an 
HSV characteristic. 
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16. The computer program product of claim 1 1 wherein said location is a computer 

screen. 

17. The computer program product of claim 1 1 wherein said 
event is the retrieval of a web page at a specified URL. 

18. The computer program product of claim 1 1 wherein said 
event is the retrieval of a data track. 

19. The computer program product of claim 1 1 wherein said object is an image residing 
in a web browser. 

20. The computer program product of claim 1 1 wherein said object is an image residing 
in a streaming media. 
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METHOD AND APPARATO FOR ASSOQATING THE 
COLOR OF AN OBJECT WITH AN EVENT 
ABSTRACT OF THE DISCLOSURE 



The present invention provides a method and apparatus for associating the color of an 
object with an event. In one or more embodiments, the invention takes a streaming or static media 
and associates an active, dynamic, or static object with an event by its given color. The event may 
be, for instance the retrieval of a document at a specified URL in a web browser or the retrieval of 
other information in non-web browser implementations. In one embodiment, the invention defines 
the color by its mathematical properties. For instance, one embodiment uses the RGB standard 
which separates the red, green, and blue properties of the color as numerical values. Another 
embodiment also uses the HSV characteristics of the color, which define hue, saturation, and value 
as components of color. Another embodiment uses both RGB and HSV characteristics. 
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